Add SciChemAgent-C1 community agent - #2
skeleton2024 wants to merge 1 commit into
Conversation
|
@xwang112358 Hi Allen, would you please review this request and score. I think public contribution agent should also provide their trajectory and solutions, rather than a single run. the source data are here: https://github.com/skeleton2024/SciChemAgent-C1/tree/main if the results look good, we can merge it. |
|
Thanks, I will look into it as soon as possible! |
|
@xwang112358 Thanks for reviewing this submission. The requested per-task C1 evidence is now published on the code repository's
SciChemAgent-C1 is deterministic and offline, so each trajectory records the public task input, the single agent execution, and the emitted solution/hash rather than hidden LLM reasoning or external tool calls. The bundle intentionally excludes scorer internals, ground truth, wall-clock timing, and machine-specific paths. The reported metrics remain local community reproductions; please use the official evaluation for acceptance and scoring. Thank you. |
|
@xwang112358 Hi Allen, just a gentle follow-up on this submission. The requested 18 solutions, 18 trajectories, and manifest have been available on the code repository main branch since August 19. Please let us know if anything else is needed for official reproduction, scoring, or merge. Thank you. |
3 participants
What changed
Adds
agentdir/SciChemAgent-C1/README.md, registering a deterministic offlineRDKit baseline for the 18 Drug Discovery C1 tasks and linking its reproducible
implementation and aggregate results.
Why
SciAgentArena describes a living benchmark with open community submissions.
SciChemAgent-C1 provides a small, auditable baseline that does not require a
hosted LLM, API keys, or network access during evaluation.
Reported result
These are community-reproduced results, not claimed as official leaderboard
scores. The linked repository contains the pinned environment, runner,
machine-readable aggregate, and independent verification script.
Validation
scripts/verify_results.py: passed; official C1 ID set matched 18/18 and thearithmetic mean recomputed to 0.9244444444
Benchmark note
The linked repository documents a pH/formal-charge inconsistency observed in
the pinned acetic-acid task instead of hard-coding the conflicting scorer value.